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In  the  first  six  and  one-half  months  of  research  supported  by  this  grant, 
we  have  begun  to  make  significant  progress  in  a  number  of  aspects  of  the 
proposed  research  areas.  First,  we  have  continued  to  work  on  problems  of 
adaptive  stochastic  control  of  queues  with  complete  observations.  This  work 
has  resulted  in  the  publication  during  this  research  period  of  the  papers  [1], 
[2]  (this  research  was  previously  supported  by  Grant  AFOSR  79-0025).  In  [1], 
the  priority  assignment  (or  dynamic  scheduling  problem)  in  a  queueing  system 
with  unknown  arrival  and  service  rate  is  considered.  The  long  term  average 
cost  criterion  with  linear  cost  rates  is  solved,  and  the  optimality  of  our 
proposed  adaptive  control  algorithm  is  shown.  A  distance-measures  approach 
to  the  problems  of  identification  and  approximation  of  queueing  systems  is 
presented  in  [2];  this  approach  combines  ideas  from  statistical  robustness, 
information-type  measures,  and  parameter-continuity  of  stochastic  processes. 

The  stochastic  adaptive  control  problems  solved  to  date  have  all,  with 
the  exception  of  those  for  linear  systems,  involved  the  assumption  of  complete 
(noiseless)  state  observations.  As  we  proposed,  we  have  begun  a  major  new 
direction  of  research  involving  adaptive  estimation  and  control  problems  for 
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stochastic  systems  with  incomplete  (or  noisy)  observations  of  the  state.  We 
have  already  been  successful  in  obtaining  some  important  new  results;  these 
are  reported  in  [3].  In  [3],  we  consider  discounted  Markov  decision  processes 
with  incomplete  state  information  and  depending  on  unknown  parameters.  The 
process  is  first  transformed  into  a  completely  observed  Markov  decision 
process  and  then  (i)  we  use  conditional  least  squares  estimation  to  obtain 
a  strongly  consistent  parameter  estimation  scheme,  and  this  is  combined  with 
a  nonstationary  value-iteration  procedure  to  obtain  (ii)  approximations 
converging  uniformly  to  the  optimal  reward  function,  and  (iii)  asymptotically 
optimal  adaptive  policies.  This  paper  provides  important  general  results,  but 
the  specific  structure  of  the  parameter  estimation  scheme  must  be  studied  in 
more  detail.  For  this  reason,  we  have  begun  an  in-depth  investigation  into 
problems  in  which  the  state  is  a  (discrete  or  continuous  time)  finite  state 
Markov  process.  Such  problems  are  discussed  in  our  proposal,  in  which  we 
proposed  to  study  convergence  of  parameter  estimates,  adaptive  estimation, 
and  adaptive  control.  This  work  is  in  progress,  and  we  have  begun  to  obtain 
preliminary  convergence  results.  This  problem  is  much  more  difficult  than 
the  linear  problems  considered  to  date,  but  our  preliminary  results  are 
encouraging. 

We  have  completed  work  in  other  areas  which  were  initially  supported 
under  Grant  AFOSR  79-0025.  In  [4],  asymptotic  approximations  for  some 
nonlinear  filtering  problems  were  derived,  analyzed,  and  compared  with  other 
filters.  Lie  algebraic  and  analytical  methods  were  utilized;  of  particular 
interest  was  the  estimation  problem  for  linear  systems  with  infrequently 
jumping  parameters.  In  [5]  and  [6],  the  structure  of  nonlinear  control 
systems  with  symmetries  was  studied,  and  the  results  obtained  were  used  to 
derive  reduced-order  algorithms  for  certain  classes  of  optimal  control 
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